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The  purpose  of  this  study  was  to  evaluate  the  teaching 
effectiveness  of  different  aspects  of  the  SCHOLAR  CAI 
system.  The  experiment  compared  how  well  students  learn 
using  SCHOLAR  with  (a)  the  interactive  map  display  of 
Map-SCHOLAR  (b)  a static  labeled  map,  and  (c)  an  unlabeled 
map.  The  results  of  the  experiment  showed  that  the  students 
learned  significantly  more  with  the  interactive  map  display, 


n 
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than  with  either  the  labeled  map  or  the  unlabeled  map.  A 
new  method  called  backtrace  analysis  was  used  to  assess  the 
effectiveness  of  specific  aspects  of  the  tutoring  strategy 
and  the  map  system  used  in  the  experiment. 
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INTRODUCTION 


In  developing  SCHOLAR,  Carbonell  (Carbonell,  1970; 
Carbonell  and  Collins,  1973)  took  a first  step  toward  a 
computer  assisted  instruction  (CAI)  system  that  is  capable 
of  conducting  general  tutorial  dialogues  with  students.  In 
SCHOLAR,  knowledge  is  not  stored  as  text,  but  in  a precisely 
structured  semantic  network  of  interrelated  facts  and 
concepts  (Quillian,  1968;  Collins  and  Quillian,  1972a). 
Every  concept  used  to  describe  a given  concept  can  itself  be 
described  elsewhere  in  the  network.  Thus,  in  a nontrivial 
sense,  the  program  can  understand  the  concepts  it  uses. 
SCHOLAR  also  has  different  subroutines  that  use  the 
structure  of  the  network  to  formulate  questions  for  the 
student,  evaluate  his  answers,  answer  his  questions,  make 
inferences  and  computations,  select  new  topics  Tor 
discussion,  etc.  The  attempt  is  to  structure  information 
like  a human  knowledge,  so  that  the  program  can  use  its 
knowledge  as  flexibly  as  a human  tutor  does. 


Because  of  its  inherent  flexibility,  the  system  can  be 
extended  in  a variety  of  ways.  For  example,  SCHOLAR  has 
been  modified  to  teach  different  kinds  of  knowledge  (Collins 
and  Grignetti,  1975)  and  to  use  different  teaching 
strategies  (Collins,  Warnock,  and  Passafiume,  1975).  Ibe 
educational  question  is  which  of  the  possible  ways  of 
varying  SCHOLAR  increase  its  teaching  effectiveness.  This 
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question  is  being  investigated  by  running  systematic 
experimental  evaluations  of  different  system  configurations 
and  different  teaching  strategies  (Collins,  197*0. 

The  experiment  reported  here  was  designed  to  test  the 
utility  of  the  map  capability  recently  added  to  the  original 
SCHOLAR  program  for  teaching  geography.  The  Map-SCHOLAR 
system  was  developed  to  integrate  the  tutoring  of  graphic 
information  with  verbal  information.  In  view  of  the 
evidence  that  pictorial  information  may  be  remembered  more 
easily  than  verbal  information,  (c.f.  Bower,  1972;  Paivio, 
1971)  we  expected  this  capability  to  increase  SCHOLAR'S 
teaching  effectiveness. 

Map-SCHOLAR  can  discuss  with  the  student  different  maps 
that  change  dynamically  according  to  the  context  of  the 
discussion.  To  do  this,  a graphic  structure  was  created 
which  parallels  the  structure  in  the  semantic  network.  The 
elements  in  the  map  display  can  be  referred  to  either  by 
their  name,  or  by  pointing  to  them,  or  both.  Map-SCHOLAR 
both  asks  and  answers  map-related  questions  and  provides 
relevant  map  information  when  the  student  makes  a mistake. 
In  short,  Map-SCHOLAR  has  all  the  capabilities  of  the 
original  SCHOLAR  with  the  addition  of  the  map  capabilities. 

Figure  1 illustrates  some  of  the  variety  of 
interactions  possible  with  Map-SCHOLAR.  The  first  panel 
illustrates  how  Map-SCHOLAR  asks  map-related  questions, 
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NOW  LET'S  SEE  WHAT  YOU  REMEMBER  ABOUT  THE  CITIES  OE  BRAZIL. 

PLEASE  NAME  THE  PRINCIPAL  CITIES  OE  BRAZIL  NOW  BLINKING 
ON  THE  DISPLAY  SCREEN. 

*MANAOS  SAO  PAULO  AND  LIMA* 

MANAOS  AND  SAO  PAULO  ARE  CORRECT.  LIMA  IS  IN  PERU,  NOT  IN  BRAZIL. 
THE  PRINCIPAL  CITIES  OF  BRAZIL  ARE  BLINKING  ON  THE  SCREEN. 

RIO  DE  JANEIRO  IS  IN  SOUTHEASTERN  BRAZIL. 

IT  IS  ON  GUANABARA  BAY  AND  THE  ATLANTIC. 

BRASILIA  IS  IN  CENTRAL  BRAZIL. 

IT  IS  FAR  FROM  THE  OCEAN. 


Fig.  1 Interactions  with  Map-SCHOLAR 


4 


LET'S  BEGIN  BY  TALKING  ABOUT  THE  TOPOGRAPHY  OF  VENEZUELA. 
THE  PRINCIPAL  LAKE  OF  VENEZUELA  IS  BLINKING,  PLEASE  NAME  IT 


: 

I 


*1  DON'T  KNOW* 

THE  PRINCIPAL  LAKE  OF  VENEZUELA  IS  BLINKING  ON  THE  SCREEN. 

LAKE  MARACAIBO  IS  IN  NORTHWESTERN  VENEZUELA. 

THE  PRINCIPAL  MINERAL  IS  OIL. 

*IS  THIS  COUNTRY  COLOMBIA?* 

YES,  THAT'S  COLOMBIA 

* WHERE  IS  THE  CAPITAL  OF  VENEZUELA?* 

CARACAS  IS  BLINKING. 


Fig.  1 Interactions  with  Map-SCHOLAR  (cont) 
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evaluates  the  student's  answers,  and  corrects  his  mistakes. 
First,  the  dots  indicating  the  location  of  the  cities  appear 
unlabeled  on  the  map  of  Brazil  and  start  blinking.  Then 
SCHOLAR  asks  the  student  to  name  the  blinking  cities.  When 
he  responds,  it  tells  him  which  answers  were  correct  and 
prints  their  names  on  the  map.  Because  he  erroneously  gave 
Lima  as  an  answer,  Map-SCHOLAR  pointed  out  that  Lima  is  in 
Peru  and  showed  where  Lima  is  on  the  map.  Thus,  Map-Scholar 
gives  both  verbal  and  visual  feedback  to  help  the  student 
learn  the  material. 

The  second  panel  shows  how  context  affects  the  map 
display.  The  topic  is  the  Amazon,  which  appears  in  this 
context  on  the  map  of  Brazil,  the  overall  topic.  The 
question  asks  the  student  to  point  at  the  cities  along  the 
Amazon  in  the  order  named.  The  dots  for  the  three  cities 
first  appear  unlabeled,  and  only  after  the  student  has 
answered  do  the  labels  appear.  Unlike  the  first  map,  this 
map  shows  only  the  cities  on  the  Amazon  and  their  relation 
to  the  Amazon.  There  is  also  expansion  of  detail  so  that  a 
city  like  Belem,  which  was  not  important  enough  to  be 
included  among  the  cities  of  Brazil,  is  included  with  the 
subtopic  of  the  Amazon.  Because  the  map  changes  dynamically 
as  the  context  changes  and  as  the  student  interacts  with  the 
system,  the  student's  attention  is  focused  on  the  relevant 
information,  and  questions  can  be  posed  in  a visual  form  not 
possible  with  a static  map  display. 
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The  third  panel  shows  the  system's  potential  for 


tutorial  interaction.  When  the  student  didn't  know  about 


Lake  Maracaibo,  Map-SCHOLAR  showed  it  on  the  screen  and 


added  the  related  verbal  information  about  the  oil  there 


This  example  also  illustrates  some  of  the  ways  in  which  the 


student  can  ask  Map-SCHOLAR  to  clarify  or  amplify  the 


information  given  (Collins  and  Warnock,  1974).  For  the 


first  question  the  student  both  pointed  at  and  named 


Colombia  to  ask  if  it  is  the  country  near  Lake  Maracaibo. 


For  the  second  question,  the  student  verbally  asked  where 


the  capital  of  Venezuela  is,  perhaps  to  find  out  how  far 


away  it  is.  SCHOLAR  figured  cut  semantically  that  the 


capital  is  Caracas,  and  then  visually  showed  where  Caracas 


is  by  blinking  it  (it  is  the  double  square).  These  examples 


illustrate  some  of  the  power  for  tutorial  interaction  that 


can  be  obtained  by  a close  integration  between  semantic  and 


visual  knowledge. 


In  order  to  test  the  utility  of  the  map  system  for 


teaching,  we  conducted  an  experiment  in  which  each  student 


learned  about  a different  country  under  one  of  three 


conditions:  one  condition  used  SCHOLAR  on  the  map  system; 


the  second  condition  used  SCHOLAR  on  a non-graphic  terminal 


but  the  student  could  look  at  a labeled  map  of  the  country; 


the  third  condition  was  like  the  second,  except  that  the 


student  was  given  an  unlabeled  map  Students'  learning  tor 


each  of  the  three  kinds  of  training  sessions  was  measured  by 


comparing  their  scores  on  a pre-test  to  those  on  a post-test 
given  three  days  after  the  last  training  session. 

A second  goal  of  thi;.  experiment  was  to  investigate  how 
specific  aspects  of  the  tutorial  dialogue  affect  students' 
learning.  To  study  this  question,  we  developed  a technique 
called  backtrace  analysis . The  technique  involves  marking 
each  piece  of  information  that  is  discussed  according  to  the 
kind  of  exchange  involved  (e.g.,  a question  requiring  a 
pointing  response  vs.  a naming  response).  By  comparing  this 
data  to  the  student's  answers  on  the  post-test,  it  is 
possible  to  identify  the  kinds  of  tutorial  interactions  that 
most  strongly  influence  the  student's  learning. 

METHOD 

Subjects . The  initial  group  of  subjects  included  nine 
high  school  students.  The  study  was  replicated  with  nine 
university  "fudents.  All  subjects  were  volunteers  and  were 
paid  for  their  services. 

Design . There  were  three  experimental  conditions:  a 
Map-SCHOLAR  condition,  a Labeled  Map  condition,  and  an 
Unlabeled  Map  condition.  The  Map-SCHOLAR  condition  was  run 
on  an  Imlac  graphic  terminal  with  the  screen  divided  between 
maps  and  verbal  communications  as  shown  in  Figure  1.  The 
student  could  input  questions  and  answers  by  a keyboard  and 
an  electronic  pointer  (a  "mouse").  The  Labeled  and 
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Unlabeled  Map  conditions  were  run  on  a keyboard  terminal 
using  a non-graphic  version  of  SCHOLAR  called  Tutor-SCHOLAR 
(Collins,  Warnock,  and  Passafiume,  1975).  The  two  versions 
of  SCHOLAR  were  identical  with  respect  to  both  teacli  lg 
strategy  and  information  in  the  data  base,  except  that 
Map-SCHOLAR  handled  all  location-related  questions  in  terms 
of  the  map,  whereas  Tutor-SCHOLAR  handled  them  verbally.  In 
the  Labeled  Map  condition,  subjects  were  given  a paper  map 
which  marked  all  the  places  (names  and  locations)  included 


in  the 

Map-SCHOLAR 

data 

base.  In  the 

Unlabeled 

Map 

condition 

, subjects 

were 

given  copies 

of  the  same 

maps , 

omitting 

the  names  of 

the 

places  that  were 

marked  on 

the 

maps  . 

For  both 

of 

these  conditions 

, students 

were 

instructed  not  to 

mark 

on  the  maps. 

The  pre- 

test , 

post-test,  and  the  final  questionnaire  were  given  in  paper 
and  pencil  format. 

Procedure . Each  student  participated  in  a preliminary 
session,  three  tutorial  sessions,  and  a post-test  session. 
The  first  purpose  of  the  preliminary  session  was  to 
administer  the  pre-test.  The  pre-test  measured  the 
student's  pre-experimental  knowledge  about  the  information 
to  be  tutored,  and  consisted  of  20  basic  questions  about  the 
geography  of  each  of  the  three  relevant  countries: 
Argentina,  Brazil,  and  Venezuela.  A secondary  purpose  of 
the  pre-test  was  to  ascertain  that  no  subject  was 
inordinately  familiar 


or  unfamiliar  with  any  one  of  these 


three  countries,  since  such  inequalities  in  prior  knowledge 
would  confound  measures  of  teaching  effectiveness.  After 
having  completed  the  pre-test,  the  student  was  given  a 
brief,  introductory  lesson  on  a fourth  country,  Chile,  using 
Map-SCHOLAR.  The  purpose  of  this  lesson  was  to  familiarize 
the  student  with  the  system  and  its  capabilities  or,  more 
specifically,  with  the  kinds  of  questions  he  would  be  asked, 
the  kinds  of  answers  that  were  expected  of  him,  the  kinds  of 
questions  he  could  ask  of  SCHOLAR,  the  use  of  the  keyboard 
and  the  pointer,  and  the  methods  by  which  he  could  correct 
his  input  errors. 


The  tutorial  phase  of  the  experiment  consisted  in 
three,  two  hour  sessions,  administered  on  consecutive  days. 
During  these  sessions,  each  student  learned  about  one 
country  in  the  Map-SCHOLAR  condition,  one  in  the  Labeled  Map 
condition,  and  one  in  the  Unlabeled  Map  condition.  Each 
lesson  lasted  for  one  hour.  After  the  student  had  received 
one  lesson  on  each  of  the  three  countries,  the  series  was 
repeated.  The  combinations  of  countries  and  teaching  modes 
were  counterbalanced  and  ordered  according  to  a 3 x 3, 
confounded,  factorial  design  (Winer,  1971,  p.  646). 

The  final  session  was  conducted  three  days  after  the 
last  tutorial  session.  In  this  session,  the  student  took 
the  post-test  and  completed  a questionnaire  on  those  aspects 
of  the  lessons  that  he  had  found  most  and  least  helpful. 
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The  post-test  was  divided  into  three  parts.  The  first  part 
consisted  of  36  basic  questions  (including  the  20  that  had 
been  on  the  pre-test)  about  each  of  the  three  countries. 
For  the  second  part  of  the  post-test,  the  student  was  given 
a map  of  each  of  the  three  countries  and  asked  to  label  the 
geographical  features  indicated.  The  third  part  of  the 
post-test  consisted  of  32  more  difficult  questions  about 
each  of  the  countries. 


Backtrace  Analysis . In  order  to  assess  the  value  of 
specific  aspects  of  the  tutorial  exchange,  we  developed  the 
technique  of  backtrace  analysis.  This  technique  involves 
marking  each  entry  in  SCHOLAR'S  database  with  respect  to  the 
way  in  which  it  is  treated  during  a given  tutorial  session. 
This  information  can  subsequently  be  retrieved,  enabling  us 
to  evaluate  the  effectiveness  of  SCHOLAR'S  various 
interactive  capabilities  from  the  probabilities  with  which 
they  result  in  correct  answers  on  the  post-test. 


More  specifically,  each  item  that  was  discussed  in  a 
given  session  was  tagged  with  information  concerning  (1)  the 
temporal  order,  (2)  the  context,  and  (3)  the  training  event 
in  which  it  arose.  For  purposes  of  the  backtrace  analysis, 
the  training  events  were  classified  as  follows: 
a)  True-False  Correct  - SCHOLAR  presents  a true-false 
question  which  the  student  answers  correctly.  SCHOLAR 
tells  the  student  he  is  correct  and  moves  on  to  new 


information  . 


m * 


b)  True-False  Error  - SCHOLAR  presents  a true-false  question 
and  the  student  answers  incorrectly  or  pleads  ignorance. 
SCHOLAR  points  out  the  correct  answer  and  goes  on. 

c)  Name  Correct  - The  student  correctly  names  a geographical 
feature(s)  in  response  to  SCHOLAR'S  request.  Each  answer 
among  a set  of  answers  is  tagged  individually.  This 
category  subsumes  what  and  where  questions  as  well  as 
f ill-in-the-blanks  and  naming  requests  by  SCHOLAR. 

d)  Name  Error  - The  student  incorrectly  names  or  fails  to 
name  a geographical  feature  when  questioned  by  SCHOLAR. 

e)  SCHOLAR  Error  Correction  - If  the  student  completes  a 
fill-in  question  erroneously,  SCHOLAR  infers  the  basis  of 
the  student's  error  and  then  presents  new  information  to 
distinguish  between  the  student's  answer  and  the  correct 
answer . 

f)  SCHOLAR  elaboration  - If  the  student  misses  a question, 
SCHOLAR  presents  related  information  at  the  same  level  of 
importance  (See  Fig.  1).  The  related  material  is  tagged 
as  an  elaboration. 

g)  Student  Question  - Information  is  introduced  as  the 
result  of  a question  the  student  asks  of  SCHOLAR. 


In  addition  to  the  above,  there  were  several  categories  of 
training  events  which  occurred  only  in  Map-SCHOLAR.  SCHOLAR 
treated  these  events  like  fill-ins,  but  they  were 
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distinctively  marked  for  purposes  of  the  backtrace  analysis 

h)  Label  - SCHOLAR  asks  the  student  to  name  those  feature 
of  the  map  that  are  blinking. 

i)  Point  - SCHOLAR  asks  the  student  to  point  to  th 
specified  geographical  features  on  the  map. 

j)  Label  and  Point  - SCHOLAR  asks  the  student  to  name  an 
point  to  a specified  set  of  geographical  features. 


s 


e 


RESULTS  AND  DISCUSSION 


The  pre-test  scores  were  examined  using  a 3 X 2 
(Countries  X Groups)  repeated  measures  analysis  of 


variance  (Winer, 

1971 , p.  518) . 

The 

only  significan 

effect  was  due 

to  groups,  as 

the 

college  student 

generally  scored 

higher  than  the 

high 

school  students 

The  number  of 

correctly  answere 

d que 

stions,  out  of  th 

possible  20  per  pre-test,  ranged  from  1 to  11 
(median  = 4.67)  for  the  college  students  and  from  0 to  5 
(median  = 0.64)  for  the  high  school  students.  Neither 
the  main  effect  of  countries  (F(2,32)  = 2.62,  p>0.05)  nor 
the  interaction  between  countries  and  groups 
(F(2,32)  = 1.36,  p>0.05)  approached  significance. 


Inasmuch  a 

s none  of  the  subjects  knew  much  about  any 

of 

the  countr 

ies  in  advance,  the  diJ' 

nee  between  pre- 

and 

post-test 

scores  should  provi  'e 

fair  estimate 

of 

SCHOLAR'S 

teaching  effectiveness. 

Moreover,  since 

the 

subjects ’ 

prior  knowledge  seemed  to 

be  evenly  distributed 
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across  countries,  the  relative  teaching  effectiveness  of 
the  three  conditions  could  be  estimated  through  direct 
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comparisons  of  the  corresponding  pre-test/post-test 


difference  scores. 


The  average  increase  in  the  number  of  correct 


responses  from  the  pre-test  to  the  post-test  is  shown  in 


Figure  2 for  each  teaching  mode.  These  difference  scores 


were  analyzed  according  to  a 3X3  (teaching  modes  X 


countries)  confounded  factorial  design.  Whereas  neither 


the  effect  of  countries  (F ( 2 , 28 ) < 1 . 0 ) nor  the  interacts 


between 


countries 


teaching 


conditions 


(F(4,28)  = 2.08,  p>0.05)  was  significant,  the  effect  of 


training  condition  was 


strongly 


significant 


(F(2,28)  = 6.05,  p<0.01 ) . According  to  a Newman-Keuls 


test  (p<0.01),  the  Map-SCHOLAR  condition  resulted  in 


significantly  higher  post-test  scores  than  the  Labeled 


Map  condition  which,  in  turn,  resulted  in  significantly 


higher  scores  than  the  Unlabeled  Map  condition, 


Separate  analyses  of  the  three  parts  of  th< 


post-test  indicated  that  much  of  the  effect  of  teaching 


modes  occurred  in  the  part  of  the  test  consisting  of  map 


labeling  questions  (F(2,28)  = 14.09,  pCO.OOl).  However, 


a pronounced  effect  of  teaching  mode  was  also  obtained 


for  the  easier,  non-map  questions  in  the  first  part  of 
the  post-test  (F ( 2 , 28 ) = 5.85,  p<0.01).  Although  the 


FIGURE 


CONDITION  DURING  TRAINING 


scores  on  the  more  difficult  questions  in  the  third  part 
of  the  post-test  were  too  noisy  to  yield  any  significant 
effects  or  interactions  under  analysis,  the  same  trend 
was  apparent.  In  short,  post-test  scores  were 
consistently  highest  in  the  Map-SCHOLAR  condition  and 
lowest  in  the  Unlabeled  Map  condition. 

These  results  indicate  that  the  map  system  not  only 
helped  students  learn  the  information  necessary  to  answer 
the  map  questions  in  Part  2 of  the  post-test,  but  also  to 
answer  the  verbal  questions  in  Parts  1 and  3 of  the 
post-test.  An  important  question  is  whether  the  benefit 
of  the  map  system  extended  only  to  verbal  information 
that  was  explicitly  about  locations,  or  whether  it  also 
extended  to  non-location  information,  such  as  the  climate 
or  terrain  of  a place.  Clearly,  one  would  expect  the  map 
system  to  help  students  learn  location  information 
better,  but  there  are  two  reasons  why  the  map  system 
might  help  students  learn  non-location  information  better 
as  well.  First,  if  map  information  showing  where  a place 
like  Manaos  is  located  helps  the  student  remember  Manaos 
better,  then  non-map  facts  about  Manaos,  such  as  its 
climate,  may  be  remembered  better.  This  is  because  the 
best  way  to  learn  something  is  to  relate  it  to 
information  already  known  (Collins  and  Quillian,  1972b; 
Norman,  1973).  Second,  if  a student  sees  that  Manaos  is 
on  the  Amazon,  then  Manaos'  climate  can  be  related  to  any 
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prior  knowledge  about  the  climate  of  the  Amazon 
(e.g.  that  the  Amazon  flows  through  jungle).  Thus,  even 
non-map  information  may  be  better  remembered  in  a visual 
context . 

This  idea  was  tested  with  backtrace  analysis  by 
separating  the  questions  during  training  into  map 
questions  and  non-map  questions,  depending  on  whether  the 
questions  were  posed  visually  by  the  map  system.  Then 
the  percentage  correct  on  the  post -test  for  the  two  types 
of  presentation  during  training  were  plotted  (see  Figure 
3).  For  map  questions,  as  expected,  students  learned 
significantly  more  with  Map-SCHOLAR  than  with  either  the 
labeled  or  unlabeled  maps.  But  for  non-map  information 
there  were  no  significant  differences,  and  students  even 
did  slightly  better  in  the  Unlabeled  Map  condition. 
Thus,  these  data  suggest  that  the  major  benefit  of  the 
map  system  is  in  learning  information  about  specific 
locations  . 

Backtrace  analysis  was  also  used  to  investigate  the 
effectiveness  of  repeating  questions,  depending  on 
whether  the  student  answers  correctly  or  incorrectly. 
Figure  4 shows  the  percentage  of  correct  responses  to 


each  item 

on 

the  post-test 

as  a function  of 

how 

frequently 

the 

students  were 

correct  or 

wrong  on 

that 

item  during 

training.  The  increases  in  the 

curves 

show 
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CONDITION  DURING  TRAINING 


QUESTIONS  CORRECT  DURING  TRAINING 


that  the  more  frequently  a student  answered  any  item 
correctly,  che  more  likely  he  was  to  recall  it  on  the 
post-test.  The  separation  of  the  curves  for  different 
numbers  of  errors  shows  that  the  more  frequently  an  item 
was  missed,  the  less  likely  it  was  to  be  recalled  on  the 
post-test.  This  simply  reflects  the  fact  that  the  items 
that  were  more  difficult  to  learn  were  likely  to  be 
missed  more  frequently.  The  concave  shape  of  the  curves 
indicates  that  the  repeated  presentation  of  a correct 
item  has  a decreasing  effectiveness.  The  implication  is 
that,  as  far  as  possible,  training  time  should  be 
allocated  to  those  items  that  the  student  has  correctly 
answered  least  often. 

When  students  missed  items  in  answering  a question, 
SCHOLAR  provided  additional  elaboration  about  some  of  the 
items  missed.  For  example,  in  Figure  1 when  the  student 


did 

not 

know  Lake 

Maracaibo,  SCHOLAR 

mentioned  the 

oil 

there 

as 

an  elaboration 

about 

Lake 

Maracaibo . 

The 

backtrace 

analysis 

showed 

that 

percent  correct  on 

the 

post-test 

increased 

from 

34$ 

when 

there  was 

no 

elaboration  of  an  item  during  training  to  4 7 % when  there 
was  one  elaboration.  This  increase  is  significant 
(t  = 4.01 ,p<0.01 ) , indicating  that  elaboration  does  help 
students  to  learn  the  material  better.  After  one 
elaboration,  the  percent  correct  stabilizes,  indicating 
that  further  elaborations  are  of  little  benefit. 


We  used  a variation  of  backtrace  analysis  to 
determine  which  kinds  of  map  questions  are  most  effective 
for  learning.  In  the  map  system  there  were  three 
different  kinds  of  map  questions  that  might  be  asked: 

(1)  pointing  questions,  where  SCHOLAR  mentioned  one  or 
more  places  and  asked  the  student  to  point  at  them 

(2)  naming  questions,  where  SCHOLAR  blinked  one  or  more 
places  and  asked  the  student  to  name  them,  and 

(3)  pointing  and  naming  questions,  where  SCHOLAR  asked 
the  student  to  name  a set  of  places,  such  as  the  rivers 
in  Brazil,  and  point  to  them  in  the  order  named. 

Figure  5 shows  the  percent  correct  on  the  second 
occurrence  of  a map  question  about  any  item  as  a function 
of  the  type  of  question  that  was  asked  on  the  first 
occurrence  of  that  i'  em . There  were  not  enough  data  for 
naming  questions,  so  they  are  not  shown.  The  column 
totals  indicate  that  students  did  better  on  pointing 
questions  than  on  pointing  and  naming  questions,  as  would 
be  expected  because  pointing  questions  are  easier. 
However,  the  row  totals  show  that  students  did  better  on 
the  second  question  if  the  first  question  had  required 
both  pointing  and  naming  than  if  it  had  required  only 
pointing  (Otl(  1 ) = 4 . 75 , p <0 . 05 ) . Evidently,  students 
learn  more  from  pointing  to  and  naming  a location  than 
from  just  pointing  to  it. 


oro>d 


CONCLUSION 


The  experiment  showed  that  students  learned 
significantly  more  with  the  interactive  map  display  than 
with  either  a static  labeled  or  unlabeled  map.  The 
advantage  of  Map-SCHOLAR  cannot  be  attributed  solely  to 
the  ability  of  the  student  to  locate  places  spatially, 
since  the  Labeled  Map  condition  allowed  the  student  to 
identify  places  just  as  effectively.  The  advantage  of 
Map-SCHOLAR  also  cannot  be  attributed  to  novelty  or  some 
other  generalized  facilitation  effect,  because,  as 
backtrace  analysis  showed,  the  effect  was  specific  to 
location  information  and  did  not  carry  over  to 
non-location  information.  The  advantage  therefore  must 
have  been  due  mainly  to  the  dynamic  aspects  of 
Map-SCHOLAR  and  its  ability  to  focus  the  student's 
attention  on  the  relevant  map  information. 

The  experiment  also  demonstrated  the  usefulness  of 
the  backtrace  analysis  technique  for  evaluating  CAI 
systems.  Backtrace  analysis  is  not  dependent  on  the  type 
of  information  being  taught,  and  is  thus  transferable  to 
CAI  systems  other  than  SCHOLAR.  Of  course,  the  specific 
tags  used  to  mark  the  data  would  change,  depending  on  the 
different  teaching  strategies  and  training  events  that 
are  being  evaluated.  The  ability  to  perform  fine-grain 
analyses  of  the  effectiveness  of  different  teaching 
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